CNN Project Part 3

• DOMAIN: Automobile

• TASK: Help to build the image dataset to be used by the AI team to build an image classifier data. Import and display the images in python against their labels. Comment on the challenges faced during this task.

(1) Import Python libraries

(1.1) Enable GPU for this notebook

In [1]:
import tensorflow as tf
print("You are using TensorFlow version", tf.__version__)
if len(tf.config.list_physical_devices('GPU')) > 0:
  print("You have a GPU enabled.")
else:
  print("Enable a GPU before running this notebook.")

from tensorflow.keras.models import Sequential  # initial NN
from tensorflow.keras.layers import Dense, Dropout, InputLayer, BatchNormalization # construct each layer
from tensorflow.keras.layers import Conv2D # swipe across the image by 1
from tensorflow.keras.layers import MaxPool2D, GlobalMaxPool2D # swipe across by pool size
from tensorflow.keras.layers import Flatten, GlobalAveragePooling2D
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.vgg16 import preprocess_input,decode_predictions

import numpy as np
import pandas as pd
import os
import cv2
import matplotlib.pyplot as plt
import matplotlib.image as mpimg 
%matplotlib inline
from tqdm.notebook import trange, tqdm
import seaborn as sns
from google.colab.patches import cv2_imshow
from IPython.display import Image, display, Markdown, clear_output

from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import LabelEncoder
from sklearn import svm, metrics
from sklearn.decomposition import PCA
from skimage import io
import math
You are using TensorFlow version 2.4.1
Enable a GPU before running this notebook.

(1.2) Mount google drive that contains image dataset

In [2]:
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).

(2) Load Image dataset

(2.1) Set the globval variables for project and image dta paths

In [3]:
image_path = '/content/drive/MyDrive/Colab Notebooks/DataFile/Cars Images/'
project_path = '/content/drive/MyDrive/Colab Notebooks/'

(2.2) Read the dataframe conatining image file names and model details

In [4]:
cars_df = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/cars_data_set.csv')
cars_df
Out[4]:
file_name category Make model segment color manufacture_year
0 00001.jpg car Audi TT sports white 2011
1 00002.jpg car Acura TL sedan black 2014
2 00003.jpg car Dodge SLT pickup truck red 2006
3 00004.jpg car Hyundai Sonata sedan red 2012
4 00005.jpg car Ford F-350 pickup truck white 2015
5 00006.jpg car Geo Metro - Convertible red 1994
6 00007.jpg car Dodge Journey suv blue 2017
7 00008.jpg car Dodge Challenger hybrid red 2020
8 00009.jpg car Mitsubishi Lancer sedan grey 2016
9 00010.jpg car Chevrolet Traverse suv grey 2012
10 00011.jpg car Buick verano sedan white 2012
11 00012.jpg car Toyota sequoia suv gold 2013
12 00013.jpg car Hyundai Elantra sedan gold 2007
13 00014.jpg car Dodge caravan minivan white 1998
14 00015.jpg car Volvo C30 sedan orange 2010
In [5]:
cars_df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 15 entries, 0 to 14
Data columns (total 7 columns):
 #   Column            Non-Null Count  Dtype 
---  ------            --------------  ----- 
 0   file_name         15 non-null     object
 1   category          15 non-null     object
 2   Make              15 non-null     object
 3   model             15 non-null     object
 4   segment           15 non-null     object
 5   color             15 non-null     object
 6   manufacture_year  15 non-null     int64 
dtypes: int64(1), object(6)
memory usage: 968.0+ bytes

Observations:

  1. There are 15 images of cars along with the make, model , segment they belong to and their manufactured year.

(2.2.1) Convert the manuafacture_year column from int to str

In [6]:
cars_df['manufacture_year'] = cars_df['manufacture_year'].astype('str')
In [7]:
cars_df['Description'] = cars_df['Make']+' '+cars_df['model'] + ' ' + cars_df['segment'] + ' ' + cars_df['color'] + ' ' + cars_df['manufacture_year']
cars_df
Out[7]:
file_name category Make model segment color manufacture_year Description
0 00001.jpg car Audi TT sports white 2011 Audi TT sports white 2011
1 00002.jpg car Acura TL sedan black 2014 Acura TL sedan black 2014
2 00003.jpg car Dodge SLT pickup truck red 2006 Dodge SLT pickup truck red 2006
3 00004.jpg car Hyundai Sonata sedan red 2012 Hyundai Sonata sedan red 2012
4 00005.jpg car Ford F-350 pickup truck white 2015 Ford F-350 pickup truck white 2015
5 00006.jpg car Geo Metro - Convertible red 1994 Geo Metro - Convertible red 1994
6 00007.jpg car Dodge Journey suv blue 2017 Dodge Journey suv blue 2017
7 00008.jpg car Dodge Challenger hybrid red 2020 Dodge Challenger hybrid red 2020
8 00009.jpg car Mitsubishi Lancer sedan grey 2016 Mitsubishi Lancer sedan grey 2016
9 00010.jpg car Chevrolet Traverse suv grey 2012 Chevrolet Traverse suv grey 2012
10 00011.jpg car Buick verano sedan white 2012 Buick verano sedan white 2012
11 00012.jpg car Toyota sequoia suv gold 2013 Toyota sequoia suv gold 2013
12 00013.jpg car Hyundai Elantra sedan gold 2007 Hyundai Elantra sedan gold 2007
13 00014.jpg car Dodge caravan minivan white 1998 Dodge caravan minivan white 1998
14 00015.jpg car Volvo C30 sedan orange 2010 Volvo C30 sedan orange 2010

(3) Visualize images

(3.1) Set the resizing image size

In [8]:
img_sz = 224

(3.2) Display all images with their labels from the dataframe.

In [9]:
cols = 4
rows = int(math.ceil(len(cars_df)/cols))
fig, ax = plt.subplots(rows,cols,figsize=(20,20))
X = []
y = []
for index in range(0,len(cars_df)):

  img = cv2.imread(image_path+cars_df['file_name'][index])
  img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
  img = cv2.resize(img,(img_sz,img_sz),interpolation = cv2.INTER_NEAREST)
  X.append(img)
  y.append(cars_df['segment'][index])


  if index < cols:
    col = index
    row = int(index/4)
  else:
    col = index % 4
    row = int(index/4)

  ax[row][col].imshow(img)
  ax[row][col].set_title(f"{cars_df['Description'][index]}",fontsize=15)
  ax[row][col].tick_params(top=False, bottom=False, left=False, right=False,labelleft=False, labelbottom=False)
plt.show()
X = np.array(X)
y = np.array(y)

(4) Use Transfer learning method to identify above cars.

Here we will use VGG16 to identify cars above

In [10]:
display(Markdown(f"#### Shape of image set is {X.shape} and {y.shape}"))

Shape of image set is (15, 224, 224, 3) and (15,)

(4.1) Use the VGG16 pre-trained model to predict the type of cars

In [11]:
from keras.applications.vgg16 import VGG16
model = VGG16(weights='imagenet')
print(model.summary())
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels.h5
553467904/553467096 [==============================] - 4s 0us/step
Model: "vgg16"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0         
_________________________________________________________________
flatten (Flatten)            (None, 25088)             0         
_________________________________________________________________
fc1 (Dense)                  (None, 4096)              102764544 
_________________________________________________________________
fc2 (Dense)                  (None, 4096)              16781312  
_________________________________________________________________
predictions (Dense)          (None, 1000)              4097000   
=================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
_________________________________________________________________
None

Observations:

  1. As we could see VGG16 has a total of 138M trainable parameters.
  2. All these parameters are pre-trained using imagenet.
  3. Also, the output layer has 1000 distint classes to choose from

(4.2) Predict the type of car from the pictures and add it to the original dataframe

In [15]:
vgg_predict = []
for img in X:
  img = np.expand_dims(img,axis=0)
  img = preprocess_input(img)
  features = model.predict(img)
  p = decode_predictions(features)
  vgg_predict.append(p[0][0][1])

cars_df['VGG16-Label'] = vgg_predict

cars_df
Out[15]:
file_name category Make model segment color manufacture_year Description VGG16-Label
0 00001.jpg car Audi TT sports white 2011 Audi TT sports white 2011 sports_car
1 00002.jpg car Acura TL sedan black 2014 Acura TL sedan black 2014 minivan
2 00003.jpg car Dodge SLT pickup truck red 2006 Dodge SLT pickup truck red 2006 pickup
3 00004.jpg car Hyundai Sonata sedan red 2012 Hyundai Sonata sedan red 2012 vacuum
4 00005.jpg car Ford F-350 pickup truck white 2015 Ford F-350 pickup truck white 2015 limousine
5 00006.jpg car Geo Metro - Convertible red 1994 Geo Metro - Convertible red 1994 mobile_home
6 00007.jpg car Dodge Journey suv blue 2017 Dodge Journey suv blue 2017 minivan
7 00008.jpg car Dodge Challenger hybrid red 2020 Dodge Challenger hybrid red 2020 sports_car
8 00009.jpg car Mitsubishi Lancer sedan grey 2016 Mitsubishi Lancer sedan grey 2016 beach_wagon
9 00010.jpg car Chevrolet Traverse suv grey 2012 Chevrolet Traverse suv grey 2012 minivan
10 00011.jpg car Buick verano sedan white 2012 Buick verano sedan white 2012 sports_car
11 00012.jpg car Toyota sequoia suv gold 2013 Toyota sequoia suv gold 2013 minivan
12 00013.jpg car Hyundai Elantra sedan gold 2007 Hyundai Elantra sedan gold 2007 limousine
13 00014.jpg car Dodge caravan minivan white 1998 Dodge caravan minivan white 1998 minivan
14 00015.jpg car Volvo C30 sedan orange 2010 Volvo C30 sedan orange 2010 tow_truck

(4.3) Visualise the images with the true label and prediction

In [19]:
cols = 4
rows = int(math.ceil(len(cars_df)/cols))
fig, ax = plt.subplots(rows,cols,figsize=(20,20))

for index in range(0,len(cars_df)):

  img = cv2.imread(image_path+cars_df['file_name'][index])
  img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
  img = cv2.resize(img,(img_sz,img_sz),interpolation = cv2.INTER_NEAREST)

  if index < cols:
    col = index
    row = int(index/4)
  else:
    col = index % 4
    row = int(index/4)

  ax[row][col].imshow(img)
  ax[row][col].set_title(f"y : {cars_df['segment'][index]} | y-hat : {cars_df['VGG16-Label'][index]}",fontsize=15)
  ax[row][col].tick_params(top=False, bottom=False, left=False, right=False,labelleft=False, labelbottom=False)
plt.show()

Observations:

  1. We could see that not all labels have been predicted correctly , especially the car Geo Metro which actually is a convertible but predicted as a mobile_home. This could be due to the reason that there is a home like structrure in the background and which dominates tha image of the car itself.

(5) Conclusion:

(5.1) Challenges faced:

  1. Cars are in general identified by their logo in the front or rear and then by their physical appearence except a few luxuary lineup. If the images do not capture this details then it would be difficult to label them.
  2. The background features most often also play a dominant role i.e if a car of interest has many other cars of different make or model in the background then the images can be potentially misidentified.
  3. There are plenty of resources in the internet but to search them by picture is challenging without a proper label or logo on the car.

(5.2) Solutions:

  1. Used Google Lens capability in our android phone to search for cars in the image dataset and get their details and this cam eto our rescue
  2. To avoid misclassification due to background images and noices, we had to crop the image of car.
In [7]:
!pwd
/content
In [9]:
%%shell
jupyter nbconvert --to html --template full /content/drive/MyDrive/Colab Notebooks/CNN_project_3.ipynb
[NbConvertApp] WARNING | pattern u'/content/drive/MyDrive/Colab' matched no files
[NbConvertApp] WARNING | pattern u'Notebooks/CNN_project_3.ipynb' matched no files
This application is used to convert notebook files (*.ipynb) to various other
formats.

WARNING: THE COMMANDLINE INTERFACE MAY CHANGE IN FUTURE RELEASES.

Options
-------

Arguments that take values are actually convenience aliases to full
Configurables, whose aliases are listed on the help line. For more information
on full configurables, see '--help-all'.

--execute
    Execute the notebook prior to export.
--allow-errors
    Continue notebook execution even if one of the cells throws an error and include the error message in the cell output (the default behaviour is to abort conversion). This flag is only relevant if '--execute' was specified, too.
--no-input
    Exclude input cells and output prompts from converted document. 
    This mode is ideal for generating code-free reports.
--stdout
    Write notebook output to stdout instead of files.
--stdin
    read a single notebook file from stdin. Write the resulting notebook with default basename 'notebook.*'
--inplace
    Run nbconvert in place, overwriting the existing notebook (only 
    relevant when converting to notebook format)
-y
    Answer yes to any questions instead of prompting.
--clear-output
    Clear output of current file and save in place, 
    overwriting the existing notebook.
--debug
    set log level to logging.DEBUG (maximize logging output)
--no-prompt
    Exclude input and output prompts from converted document.
--generate-config
    generate default config file
--nbformat=<Enum> (NotebookExporter.nbformat_version)
    Default: 4
    Choices: [1, 2, 3, 4]
    The nbformat version to write. Use this to downgrade notebooks.
--output-dir=<Unicode> (FilesWriter.build_directory)
    Default: ''
    Directory to write output(s) to. Defaults to output to the directory of each
    notebook. To recover previous default behaviour (outputting to the current
    working directory) use . as the flag value.
--writer=<DottedObjectName> (NbConvertApp.writer_class)
    Default: 'FilesWriter'
    Writer class used to write the  results of the conversion
--log-level=<Enum> (Application.log_level)
    Default: 30
    Choices: (0, 10, 20, 30, 40, 50, 'DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL')
    Set the log level by value or name.
--reveal-prefix=<Unicode> (SlidesExporter.reveal_url_prefix)
    Default: u''
    The URL prefix for reveal.js (version 3.x). This defaults to the reveal CDN,
    but can be any url pointing to a copy  of reveal.js.
    For speaker notes to work, this must be a relative path to a local  copy of
    reveal.js: e.g., "reveal.js".
    If a relative path is given, it must be a subdirectory of the current
    directory (from which the server is run).
    See the usage documentation
    (https://nbconvert.readthedocs.io/en/latest/usage.html#reveal-js-html-
    slideshow) for more details.
--to=<Unicode> (NbConvertApp.export_format)
    Default: 'html'
    The export format to be used, either one of the built-in formats
    ['asciidoc', 'custom', 'html', 'latex', 'markdown', 'notebook', 'pdf',
    'python', 'rst', 'script', 'slides'] or a dotted object name that represents
    the import path for an `Exporter` class
--template=<Unicode> (TemplateExporter.template_file)
    Default: u''
    Name of the template file to use
--output=<Unicode> (NbConvertApp.output_base)
    Default: ''
    overwrite base name use for output files. can only be used when converting
    one notebook at a time.
--post=<DottedOrNone> (NbConvertApp.postprocessor_class)
    Default: u''
    PostProcessor class used to write the results of the conversion
--config=<Unicode> (JupyterApp.config_file)
    Default: u''
    Full path of a config file.

To see all available configurables, use `--help-all`

Examples
--------

    The simplest way to use nbconvert is
    
    > jupyter nbconvert mynotebook.ipynb
    
    which will convert mynotebook.ipynb to the default format (probably HTML).
    
    You can specify the export format with `--to`.
    Options include ['asciidoc', 'custom', 'html', 'latex', 'markdown', 'notebook', 'pdf', 'python', 'rst', 'script', 'slides'].
    
    > jupyter nbconvert --to latex mynotebook.ipynb
    
    Both HTML and LaTeX support multiple output templates. LaTeX includes
    'base', 'article' and 'report'.  HTML includes 'basic' and 'full'. You
    can specify the flavor of the format used.
    
    > jupyter nbconvert --to html --template basic mynotebook.ipynb
    
    You can also pipe the output to stdout, rather than a file
    
    > jupyter nbconvert mynotebook.ipynb --stdout
    
    PDF is generated via latex
    
    > jupyter nbconvert mynotebook.ipynb --to pdf
    
    You can get (and serve) a Reveal.js-powered slideshow
    
    > jupyter nbconvert myslides.ipynb --to slides --post serve
    
    Multiple notebooks can be given at the command line in a couple of 
    different ways:
    
    > jupyter nbconvert notebook*.ipynb
    > jupyter nbconvert notebook1.ipynb notebook2.ipynb
    
    or you can specify the notebooks list in a config file, containing::
    
        c.NbConvertApp.notebooks = ["my_notebook.ipynb"]
    
    > jupyter nbconvert --config mycfg.py

---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-9-2817b50e4a3d> in <module>()
----> 1 get_ipython().run_cell_magic('shell', '', 'jupyter nbconvert --to html --template full /content/drive/MyDrive/Colab Notebooks/CNN_project_3.ipynb')

/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2115             magic_arg_s = self.var_expand(line, stack_depth)
   2116             with self.builtin_trap:
-> 2117                 result = fn(magic_arg_s, cell)
   2118             return result
   2119 

/usr/local/lib/python3.7/dist-packages/google/colab/_system_commands.py in _shell_cell_magic(args, cmd)
    111   result = _run_command(cmd, clear_streamed_output=False)
    112   if not parsed_args.ignore_errors:
--> 113     result.check_returncode()
    114   return result
    115 

/usr/local/lib/python3.7/dist-packages/google/colab/_system_commands.py in check_returncode(self)
    137     if self.returncode:
    138       raise subprocess.CalledProcessError(
--> 139           returncode=self.returncode, cmd=self.args, output=self.output)
    140 
    141   def _repr_pretty_(self, p, cycle):  # pylint:disable=unused-argument

CalledProcessError: Command 'jupyter nbconvert --to html --template full /content/drive/MyDrive/Colab Notebooks/CNN_project_3.ipynb' returned non-zero exit status 255.
In [ ]:

In [ ]: